Electronic computing has evolved from primitive, vacuum-tube-based computer systems, initially developed during the 1940s, to modern electronic computing systems in which large numbers of multi-processor computer systems, such as server computers, work stations, and other individual computing systems are networked together with large-capacity data-storage devices and other electronic devices to produce geographically distributed computing systems with hundreds of thousands, millions, or more components that provide enormous computational bandwidths and data-storage capacities. These large, distributed computing systems are made possible by advances in computer networking, distributed operating systems and applications, data-storage appliances, computer hardware, and software technologies.
In modern computing systems, individual computers, subsystems, applications, operating systems, and components generally output large volumes of status, informational, and error messages that are collectively referred to as “log messages.” In large, distributed computing systems, terabytes of log messages may be generated each day. The log messages are typically sent to a log management server that records the log messages in event logs stored in data-storage appliances. Log management tools also apply a series of tagging operations to tokens in each log message, which is an extremely computationally intensive and time-consuming process. Because the volume and rate at which log messages are received at a log management server are so high, the log messages are typically buffered so that the series of tagging operations can be applied to each log message. Subjecting each log message to the full series of tagging operations increases cost of storage and creates a significant delay in the amount it takes for real-time analysis of log messages for potential problems. System administrators seek methods and systems that reduce the amount of temporary data storage for ingesting log messages and reduces the amount of time between ingesting log messages and recording the log messages in an event log.